Analysis of missing data patterns

Columbia University

Paris School of Economics

In this document, we analyse missing data patterns in the dataset.

First of all, we noticed that there are some issues with the dataset and therefore, for now, restrict the dataset to get rid off these issues. In particular, we do not have any pollution data for 2014 nor for NO in 2015 and very few observations for PMs in 2019. These issue do not come from actual missing data but are due to issue with the package we used to access the pollution data. These data are available through the EEA interface.

data <- data_imputation %>%  
  mutate(year = lubridate::year(date)) %>% 
  filter(year != 2014) %>% 
  filter(!(year == 2015 & pollutant == "no")) %>% 
  filter(!(year == 2019 & str_sub(pollutant, 1, 2) == "pm")) %>% 
  select(-year)
  
# rm(data_imputation)

Location of measurement stations

The stations considered are located in the 17 biggest largest in France:

City	Number of stations
Bordeaux	4
Clermont-Ferrand	13
Dijon	7
Grenoble	5
Le Havre	17
Lille	7
Lyon	22
Marseille	11
Montpellier	7
Nancy	4
Nantes	16
Nice	9
Paris	29
Rennes	9
Rouen	7
Strasbourg	9
Toulouse	19

Missing data patterns in the sample

Here, we investigate whether missing pollutant concentration data varies across different dimentions.

The overall share of missing values is 0.046308.

#' Graphs describing the proportion of missing values in different groups
#'
#' Enables to plot the proportion of missing values by a number of 
#'
#' @param df A dataframe. Dataframe to analyse 
#' @param grouping_var A string. Name of the variable, in \code{df}, for which the proportion of missing values needs to be displayed
#' @param bins An integer. Number of bins to slice \code{grouping_var} in, if it is a continuous value.
#' @param facet_var A string. Name of the variable, in \code{df}, by which to facet the initial graph
#' @param y_axis_to An integer. Limit of the y axis
#' @export
#'
hist_missing_by <- function(df, variable_missing = "concentration", grouping_var, bins = 10, facet_var = NA, y_axis_to = 1) {
  
  grouping_var_name <- str_replace_all(grouping_var, pattern = "_", replacement = " ")
  variable_missing_name <- str_replace_all(variable_missing, pattern = "_", replacement = " ")
  
  if (!is.factor(df[[grouping_var]]) && !is.na(as.numeric(df[[grouping_var]]))) {
    df[[grouping_var]] <- cut_interval(as.numeric(df[[grouping_var]]), bins)
  } 
  
  graph <- df %>% 
    filter(!is.na(.data[[grouping_var]])) %>% 
    group_by_(grouping_var, facet_var) %>%
    summarise(share_missing = sum(is.na(.data[[variable_missing]]))/n()) %>%
    arrange(share_missing) %>%
    ggplot() +
    # geom_point(aes(x = .data[[grouping_var]], y = share_missing)) +
    # geom_line(aes(x = .data[[grouping_var]], y = share_missing)) +
    geom_col(aes(x = .data[[grouping_var]], y = share_missing)) +
    labs(
      title = paste("Share of missing", variable_missing_name,"data by", grouping_var_name),
      x = str_to_sentence(grouping_var_name), 
      y = paste("Share of missing", variable_missing_name,"data")
    ) 
  
  if (!is.null(y_axis_to)) {
    graph <- graph + 
      ylim(0, y_axis_to) 
  }
   
  if (!is.na(facet_var)) {
    graph <- graph +
    facet_wrap(facets = facet_var)
  } 
  
  return(graph)
}

Across pollutants

We can break this down by pollutant.

One can notice that the share of missing values varies across pollutants, up to about a factor two. This higlights the potential necessity of analysing missingness patterns independently across pollutants.

Across locations

We look weather missingness patterns vary across cities.

data %>% 
  hist_missing_by(grouping_var = "city") +
  coord_flip()

data %>%  
  hist_missing_by(grouping_var = "site_area") 

data %>%  
  hist_missing_by(grouping_var = "site_type") 

data %>% 
  hist_missing_by(grouping_var = "elevation") 

data %>% 
  hist_missing_by(grouping_var = "latitude") 

Across dates and time

Overall, we notice that, along the time dimension data almost seems to be missing at random appart from a decreasing trend in the proportion of missing data

data %>%  
  mutate(year = year(date)) %>% 
  hist_missing_by(grouping_var = "year") 

We look at whether this variation comes more from some pollutants than others.

data %>% 
  mutate(year = year(date)) %>% 
  group_by(pollutant, year) %>% 
  summarise(share_missing = sum(is.na(concentration))/n()) %>% 
  arrange(share_missing) %>% 
  ggplot() + 
  geom_col(aes(x = factor(year), y = share_missing, fill = pollutant), position = "dodge") + 
  labs(title = "Share of missing PM10 concentration data by year and pollutant", x = "Year", y =  "Share of missing PM10 concentration data") +
  ylim(0, 1)

Zooming in and looking at monthly data, we can see that this decreasing trend is somehow step wise, with some important decreases between December and January.

data %>%  
  mutate(year_month = paste(year(date), str_pad(month(date), 2, pad = 0))) %>% 
  hist_missing_by(grouping_var = "year_month") 

We now investigate whether the share of missing data varies across month of the year, day of the month and day of the week. There does not seem to be huge variations.

Month

data %>%
  mutate(month = factor(month(date))) %>% 
  hist_missing_by(grouping_var = "month")

data %>%  
  mutate(
    year = factor(year(date)), 
    month = factor(month(date))
  ) %>% 
  hist_missing_by(grouping_var = "month", facet_var = "year")

Day of the month

data %>%
  mutate(day_of_month = factor(day(date))) %>% 
  hist_missing_by(grouping_var = "day_of_month")

data %>%  
  mutate(
    year = factor(year(date)), 
    day_of_month = factor(day(date))
  ) %>% 
  hist_missing_by(grouping_var = "day_of_month", facet_var = "year")

data %>%  
  mutate(
    month = factor(month(date)), 
    day_of_month = factor(day(date))
  ) %>% 
  hist_missing_by(grouping_var = "day_of_month", facet_var = "month")

Day of the week

data %>%
  mutate(day_in_week = factor(wday(date))) %>% 
  hist_missing_by(grouping_var = "day_in_week")

data %>%  
  mutate(
    year = factor(year(date)), 
    day_in_week = factor(wday(date))
  ) %>% 
  hist_missing_by(grouping_var = "day_in_week", facet_var = "year")

data %>%  
  mutate(
    month = factor(month(date)), 
    day_in_week = factor(wday(date))
  ) %>% 
  hist_missing_by(grouping_var = "day_in_week", facet_var = "month")

Hour of the day

Now we investigate whether there are some variation in missingness patterns across time.

data %>%
  mutate(hour = factor(hour(date))) %>% 
  hist_missing_by(grouping_var = "hour")

data %>%
  mutate(hour = factor(hour(date))) %>% 
  hist_missing_by(grouping_var = "hour", facet_var = "pollutant")

Across weather patterns

Temperature

data %>%
  hist_missing_by(grouping_var = "temperature")

Wind

data %>% 
  hist_missing_by(grouping_var = "wind_speed")

data %>% 
  hist_missing_by(grouping_var = "wind_speed_max")

data %>% 
  hist_missing_by(grouping_var = "wind_direction", y_axis_to = 0.1) +
  coord_polar()

data %>% 
  hist_missing_by(grouping_var = "wind_direction_max", y_axis_to = 0.1) +
  coord_polar()

Rainfall

data %>% 
  hist_missing_by(grouping_var = "rainfall_height") 

data %>% 
  hist_missing_by(grouping_var = "rainfall_duration") 

Other weather variables

data %>% 
  hist_missing_by(grouping_var = "relative_humidity") 

data %>% 
  hist_missing_by(grouping_var = "sea_level_pressure") 

data %>% 
  hist_missing_by(grouping_var = "insolation_duration") 

data %>% 
  hist_missing_by(grouping_var = "global_radiation") 

data %>% 
  hist_missing_by(grouping_var = "uv_radiation")